Papers with adaptation process
Parameter-efficient Tuning for Large Language Model without Calculating Its Gradients (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent parameter-efficient tuning methods can only save 30% of training memory . gradient computation and backpropagation are still necessary for these methods . |
| Approach: | They propose a parameter-efficient tuning method that can be used to fine-tune large language models without calculating gradients. |
| Outcome: | The proposed method saves 30% of training memory and improves performance on large language models. |
ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular method . however, it is implemented with a fixed intrinsic rank that might not be ideal for downstream tasks. |
| Approach: | They propose a method that estimates the importance score of each LoRA rank and prunes abundant LoRA ranks to improve performance. |
| Outcome: | The proposed method outperforms baselines on a variety of tasks with comparable parameters. |
Breaking Boundaries in Retrieval Systems: Unsupervised Domain Adaptation with Denoise-Finetuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain adaptation methods for dense retrieval models use unadapted rerank models, leading to imprecise labels. |
| Approach: | They propose to adapt a rerank model to the target domain before using it for label generation. |
| Outcome: | The proposed model achieves better results across three retrieval datasets. |
MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning (2024.acl-long)
Copied to clipboard
Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, Jiahuan Pei
| Challenge: | Large language models (LLMs) are the default paradigm for natural language processing (NLP) as the models’ scale and the diversity of tasks increase, fine-tuning becomes infeasible. |
| Approach: | They propose to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters and reduce their rank by 8 times . |
| Outcome: | The proposed model uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential. |
Sparse Low-rank Adaptation of Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained large language models in a parameter-efficient manner are gaining traction within the research community. |
| Approach: | They propose a method of low-rank adaptation that enables dynamic adjustments to the intrinsic rank during the adaptation process. |
| Outcome: | The proposed approach outperforms the current method with a fixed and unalterable intrinsic rank and a low-rank adaptation process. |
Emotion-Anchored Contrastive Learning Framework for Emotion Recognition in Conversation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Emotion Recognition in Conversation (ERC) is a task that aims to identify the emotions behind each utterance in a conversation. |
| Approach: | They propose an Emotion-Anchored Contrastive Learning framework that generates more distinguishable utterance representations for similar emotions. |
| Outcome: | The proposed framework achieves state-of-the-art on similar emotions and performs well on similar ones. |
The Hidden Space of Transformer Language Adapters (2024.acl-long)
Copied to clipboard
| Challenge: | Adapters are small modules trained on top of a frozen language model to adapt predictions to new target languages. |
| Approach: | They propose to train transformer language adapters on top of a frozen model to adapt predictions to new target languages. |
| Outcome: | The transformer language adapters are trained on top of a frozen model to adapt predictions to new target languages. |
MELT: Materials-aware Continued Pre-training for Language Model Adaptation to Materials Science (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focused on constructing domain-specific corpus focus on a limited and scarce nature of datasets in materials science poses significant challenges for developing models that generalize well across a broad range of materials entities. |
| Approach: | They propose a method to adapt pre-trained language models for materials science by continuously pre-training them on a materials science corpus. |
| Outcome: | The proposed method is able to adapt pre-trained language models for materials science tasks. |
Complex Logical Query Answering by Calibrating Knowledge Graph Completion Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to solve complex logical queries are not well-calibrated . CKGC is lightweight and effective, allowing the model to quickly converge . |
| Approach: | They propose a method for calibrating KGC models to adapt to complex logical queries . they map the values of predictions of KGC to the range [0, 1] . |
| Outcome: | The proposed method can significantly boost model performance in complex logical query answering task. |
Towards Cross-Cultural Machine Translation with Retrieval-Augmented Generation from Multilingual Knowledge Graphs (2024.emnlp-main)
Copied to clipboard
| Challenge: | XC-Translate is a large-scale, manually-created benchmark for machine translation . current systems struggle to translate texts containing entity names, but KG-MT outperforms state-of-the-art approaches . |
| Approach: | They propose a method to integrate multilingual knowledge into a neural machine translation model . XC-Translate is the first large-scale, manually-created benchmark for machine translation . they propose KG-MT to integrate cultural-related references into MT models . |
| Outcome: | The proposed method outperforms state-of-the-art approaches by a large margin compared to NLLB-200 and GPT-4 . the proposed method is based on a multilingual knowledge graph and dense retrieval mechanism . |
Analyzing Film Adaptation through Narrative Alignment (2023.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines the book-to-film adaptation process by examining the differences between the two media . novel adaptations often require dropping sections of the source text from the movie script . |
| Approach: | They use a Smith-Waterman local alignment algorithm to quantify text similarity between scenes and book units. |
| Outcome: | The proposed method reveals that novel adaptations often require dropping parts of the source text from the movie script. |
Context-Aware Adapter Tuning for Few-Shot Relation Learning in Knowledge Graphs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to predict instances for missing relations on knowledge graphs are limited by their limited training examples. |
| Approach: | They propose a context-aware adapter for few-shot relation learning in KGs . they propose tunable relation adaptation and contextual information for each relation . |
| Outcome: | Experiments on three benchmark KGs validate the superiority of RelAdapter over state-of-the-art methods. |
STAF: Pushing the Boundaries of Test-Time Adaptation towards Practical Noise Scenarios (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained language models have demonstrated superior performance on NLP tasks . however, when the training domain and testing domain are taken from different distributions, the deployed model often violates this assumption. |
| Approach: | They propose a Stable Test-time Adaptation Framework to stabilize the adaptation process. |
| Outcome: | The proposed framework boosts model robustness to noise distribution shifts while minimizing error accumulation and catastrophic forgetting. |
SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptation (2026.acl-long)
Copied to clipboard
Mahi Luthra, Jiayi Shen, Maxime Poli, Angelo Ortiz Tandazo, Yosuke Higuchi, Youssef Benchekroun, Martin Gleize, Charles-Éric Saint-James, Dongyan Lin, Phillip Rust, Angel Villar-Corrales, null Surya, Vanessa Stark, Rashel Moritz, Juan Pino, Yann LeCun, Emmanuel Dupoux
| Challenge: | Empirically, SpidR-Adapt achieves rapid gains in phonemic discriminability and downstream spoken language modeling scores . current self-supervised learning models require thousands of hours of training data to learn meaningful linguistic representations. |
| Approach: | They propose a bi-level optimization framework for rapid adaptation of speech units to new languages using minimal unlabeled data. |
| Outcome: | The proposed model achieves rapid gains in phonemic discriminability and spoken language modeling scores . it surpasses in-domain toplines after training on less than 1h of target-language audio . |